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M.  I.  1.  LioKArtiES 


Dynamics  for  Establishing  the  Flows  of  Information 
Needed  for  Effective  Planning.  Administration  and  Control. 


ABSTRACT 
Many  studies  have  described  the  life  cycle  of  an  existing  management 
information  systems  and  also  have  prescribed  what  it  should  be.   Although 
there  are  some  differences  in  terminology  and  in  detail,  it  seems  to  be 
broad  consensus  that  one  of  the  first  steps  is  to  determine  the  informa- 
tion requirements  of  the  users  and  to  quantify  what  benefits  can  be 
expected  to  accrue  from  filling  these  needs  and  also  the  costs  to  be 
incurred.   This  need,  however,  for  design  of  a  system  to  determine  and 
evaluate  the  information  requirements  of  managers  has  been  overshadowed 
by  the  technical  problems  of  hardware-software  design  and  optimization. 
To  redress  this  imbalance  the  DEFINEPAC  system  has  been  designed.   This 
paper  reports  some  early  research  results  of  the  DEFINEPAC  system.   The 
system  tested  a  design  for  ascertaining  and  evaluating  the  information 
requirements  for  ill-structured  and  "wicked"  problems  of  "Management 
Planning  and  Control."  In  the  pilot  study  the  system  was  17%  more  effec- 
tive in  ascertaining  information  requirements  than  classical  method. 
Moreover,  parallel  application  of  the  system  to  several  managers  having 
like  responsibilities  permitted  a  further  19%  improvement. 
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Dynamics  for  Establishing  the  Flows  of  Information 
Needed  for  Effective  Planning.  Administration  and  Control 


1.   INTRODUCTION 

Many  studies  [4.5,6,10,]  have  described  how  the  life  cycle  of  a 
management  information  system  does  or  should  proceed.   Although  there 
are  slight  differences  in  terminology  and  in  detail  there  seems  to  be 
broad  consensus  that  one  of  the  first  steps  is  to  clarify  exactly  what 
are  the  requirements  of  users  and  to  quantify  what  benefits  can  be 
expected  to  accrue  and  costs  to  be  incurred  from  filling  these  needs. 

A  literature  search  convinced  us  that  the  research  resources 
devoted  to  the  various  steps  of  technical  systems  design  and  implementa- 
tion (which  follow  this  initial  requirement's  analysis),  vastly  exceed 
those  committed  to  trying  to  improve  the  processed  whereby  user  needs 
are  perceived,  specified  and  evaluated.   By  the  dame  token  we  found 
that  managers  tend  to  complain  far  more  about  the  Ineffectiveness  of 
their  MIS  support  than  about  its  inefficiency.   This  complaint,  more- 
over, has  been  fairly  well  documented  [19]. 

On  this  basis  we  launched,  in  1972.  a  concerted  research  effort 
to  improve  the  tools  available  for  use  in  this  important  but  under- 
researched  phase  of  systems  analysis.   Four  major  problem  areas  were 
soon  found  to  be  of  paramount  importance: 

1.  Techniques  for  documenting  requirements  in  a  way 
which  would  not  preempt  critical  design  choices  appro- 
priately part  of  the  implementation  process. 

2.  Techniques  for  quantifying  the  benefits  which  would 
accrue  if  suggested  system  improvements  were  implemented. 


(Such  evaluations  are  essential  to  the  subsequent  processes 
of  cost-benefit  analysis,  project  selection  and — upon  accep- 
tance— project  prioritization) . 

3.  Techniques  for  providing  overall  structure  to  the 
requirements  study  itself. 

4.  Techniques  for  managing  the  details  of  analyst-user 
interviews,  to  improve  communication  and  thereby  raise 
the  probability  that  all  needs  would  be  discovered  and 
accurately  described. 

The  first  of  these  problems  (the  appropriate  problem  statement 
language)  has  been  the  subject  of  intensive  research  in  project  ISDOS 
and  elsewhere.   These  results  are  well  summarized  by  Teichroew  [27]. 

The  second  of  these  problems  (practical  information  evaluation)  is 
the  subject  of  an  extensive  research  effort  we  are  presently  conducting. 
The  preliminary  highlights  from  that  study  are  briefly  reviewed  in 
section  3  of  this  paper  because  they  are  pre-requisite  to  an  under- 
standing of  the  research  findings  we  wish  to  report  in  section  5. 

The  third  of  these  problems  (the  overall  structuring  of  the 
information  requirements  analysis)  has  been  under-researched  [15,18] 
and  is  presently  the  subject  of  further  research  involving  the  field- 
testing  of  a  radically  improved  version  of  those  procedures.   But 
any  significant  gains  in  the  overall  management  of  projects  intended 
to  analyze  information  requirements  must  begin  with  a  better  under- 
standing of  the  detailed  processes  by  which  individual  analysts  can 
go  about  the  task  of  learning  the  needs  of  individual  users.   Under- 
standing the  building-block,  is  pre-requisite  to  designing  the  wall. 
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This,  then,  is  the  subject  of  the  research  we  wish  to  report  here. 

In  the  next  section  we  address  and  reconcile  the  fundamental  dilemma 
which  has  hitherto  bedevilled  attempts  to  structure  both  the  total  require- 
ments analysis  process  and  its  detailed  user  interview  components.   In 
section  3  we  delineate  the  practical  approach  to  quantifying  prospective 
benefits  used  in  our  study.   In  section  4  we  describe  the  pilot  study 
itself.   And  in  section  5  we  review  its  detailed  findings.   The  con- 
clusions which  they  support  may  be  briefly  summarized  as  follows: 

1.  It  is,  indeed,  true  that  simply  asking  users  what  they 
need  is  an  extremely  inefficient  and  ineffective  way  of 
discovering  their  requirements. 

2.  It  is,  indeed,  possible  to  adumbrate  a  model  to  help 
give  form  to  the  analysis  of  ill-structured  problems  without 
forcing  them  into  a  simplistic  and  pre-conceived  mold.   Such 

a  model,  however,  will  help  trigger  the  memory  and  aid  users  in 
organizing  the  specifications  of  their  information  require- 
ments. 

3.  It  is,  indeed,  possible  to  have  users  place  a  value  on 
alternative  MIS  service  levels  despite  the  fuzziness  of  their 
problems  and  the  decision  procedures  they  follow. 

4.  In  interviewing  decision-makers,  worthwhile  improvements 
can  be  gained  by  interviewing  several  people  about  the  same 
topic;  though  there  would  be  considerable  overlap  between 
their  suggestions  but  what  one  forgets  another  may  well 
recall  and  elaborate,  and  the  process  can  be  justified  at  a 
certain  level  of  cost. 


2.   THE  FUNDAMENTAL  DILEMMA  OF  INFORMATION  REQUIREMENTS  STUDIES. 

There  is  a  great  deal  of  variety  in  the  ways  analysts  go  about 
the  business  (macro  and  micro)  of  determining  information  requirements. 
Miller  [20]  makes  a  fundamental  distinction  between  those  who  ask  their 
clients  (his  "research"  approach)  and  those  who  employ  "expertise"  and 
tell  them  what  they  need.   As  we  shall  see.  the  matter  at  issue  here 
on  the  micro  level  is  essentially  the  same  as  that  behind  the  macro  level 
controversy  as  to  whether  requirements  studies  should  proceed  "top  down" 
or  "bottom  up"  through  the  organization. 

During  the  first  phase  of  their  development,  data  processing 
systems  were  primarily  concerned  with  achieving  cost  reductions 
through  clerical  savings.   During  this  phase  the  expertise  of  the 
methods  and  procedures  analyst  was  an  effective  source  of  requirements 
specification. 

More  recently  systems  improvements  have  been  addressed  primarily 
to  the  area  of  operational  control—production  scheduling,  routing 
problems  and  other  logistic  applications  such  as  inventory  control. 
During  this  phase  the  requirements  specifications  have  generally  ori- 
ginated from  the  expertise  of  analysts  familiar  with  the  models  and 
decision  algorithms  of  management  science. 

But  today  the  potential  gains  from  developments  of  these  earlier 
types  are  very  largely  exhausted  and  now  there  is  a  pressing  need  to 
supply  better  information  systems  support  to  those  more  senior  levels  of 

management  with  responsibility  for  ill-structured  problems.* 

*Sprague  [25],  following  Gorry  and  Morton  [14],  points  out  that  ill- 
structured  problems  can  also  be  encountered  at  fairly  low  levels  of 

management. 
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And  it  is  here  that  we  encounter  a  fundamental  dilemma.   On  the 
one  hand,  as  Ackoff  [1]  has  pointed  out,  simply  "asking"  these  managers 
what  they  require  will  not  work  because  the  underlying  assumption  that, 
unaided,  they  know  their  own  information  needs  is  demonstrably  false 
(the  reservation  of  Rappoport  [22],  notwithstanding).   But  on  the  other 
hand  there  is  no  ready  and  obvious  way  to  use  "expertise"  to  specify 
their  needs  for  them  since,  by  definition,  ill-structured  problems  are 
amenable  neither  to  modelling  nor  to  algorithmic  solution. 

A  number  of  attempts  have  been  made  to  resolve  this  dilemma  but 
none  seems  altogether  satisfactory.  Orden  [21]  and  Chadler  and  Nador  [9] 
take  what  is  essentially  the  same  approach  as  the  old  methods  and  pro- 
cedures experts  (though  in  modern  guise).   They  base  their  process  on 
the  analysis  of  present  forms  and  reports.   We  have  shown  elsewhere  [16] 
that  effective  reporting  design  must  be  based  on  the  purpose  the  reports 
are  to  serve  rather  than  the  procedures  for  generating  them;  we  would 
also  stress  that  reports  specification  is  only  a  very  small  sub-set  of 
MIS  requirements  analysis.   Taggart  [26]  bases  his  approach  upon  a 
textual  analysis  of  job  descriptions;  but  these  are  notoriously  unsatis- 
factory and,  at  any  rate,  give  only  the  sketchiest  of  clues  as  to  what 
information  is  needed  to  carry  out  these  responsibilities  effectively. 
Burns  [8]  has  used  what  is  essentially  a  time-diary  approach;  but  this 
overlooks  those  problems  which  are  important  but  infrequent  and  again 
describes  responsibilities  rather  than  the  essential  issue,  the  informa- 
tion needed  to  cope  with  them  effectively.   Blumenthal  [6],  Zani  [28]  and 


Glans  et.al.  [13]  base  their  approach  upon  a  "top  down"  chain  of  deduc- 
tive means-end  inference  from  the  goals,  objectives  and  major  programs 
of  the  organization;  but  this  approach  is  so  vague  and  general  that  it 
is  tantamount  to  simply  asking  executives  what  they  want.   On  the  other 
hand  the  operations  research  approach  is  to  use  a  decision  model  prefer- 
ably mathematical,  and  then  derive  information  on  parameter  values. 
However,  this  approach  is  very  unsatisfactory  for  unstructured  problems. 

By  contrast,  we  assume  that  effective  inquiry  requires  a  structured 
set  of  cues  to  trigger  memory  and  to  jEocus  managerial  attention.   Unstruc- 
tured inquiry  may  elicit  good  suggestions  but  these  will  be  so  far  from 
exhaustive  that  the  resulting  MIS  will  be  of  marginal  value.   The  dilemma 
we  must  resolve,  then,  is  to  model  the  "unmodellable" . 

Conventionally  the  quality  of  a  model  is  judged  primarily  by  some 
measure  of  correlation  between  predicted  and  actual  behavior;  and  parsi- 
mony as  a  secondary  consideration.   The  normal  approach  to  modelling 
extremely  complex  systems  is  to  restrict  the  variety  of  performance 
characteristics  which  the  model  seeks  to  predict  whether  by  excluding 
"irrelevant"  variables  from  attention  or  by  use  of  homomorphic  aggraga- 
tions  (an  extreme  case  is  Keynes'  [17]  four-variable  model  of  the  economy). 
To  apply  this  philosophy  to  the  study  of  information  requirements  for 
ill-structured  problem  solving,  however,  is  extremely  perilous  for  there 
is  every  likelihood  that  important  variables  will  not  only  be  overlooked 
by  the  model  but  that  the  ommission  will  pass  unnoticed  because  the  model- 
as  a  self-fulfilling  prophecy — has  implicitly  declared  them  "irrelevant". 


The  last  thing  the  requirements  analyst  can  afford  is  to  be  parsimonious 
in  his  survey  of  possibly  valuable  information. 

Economize  he  must,  however,  if  his  study  is  to  be  kept  within  manag- 
able  bounds.   The  approach  we  take,  therefore,  is  to  minimize  the  effort 
devoted  to  specifying  the  relationships  between  variables.  Normally  a 
model  requires  closed-form  functional  specification  of  all  relationships. 
Economists,  using  the  "comparative  statics"  approach,  have  been  success- 
ful in  reducing  this  requirement — ^without  total  loss  of  utility — to  mere 
specification  of  the  signs  of  all  partial  derivatives.   We  propose  that 
for  information  requirements  analysis  it  is  possible  and  desirable  to  go 
yet  one  step  further:   all  that  is  needed  is  an  incidence  matrix  indicat- 
ing which  input  variables  are  (somehow)  relevant  to  decisions  about 
output  variables. 

Following  this  philosophy,  we  have  derived  the  model  illustrated  in 
fig.  1  as  a  general  but  unrestrictive  framework  for  our  interview  protocol. 
Given  a  knowledge  of  the  activities  and  resources  for  which  a  manager  is 
responsible,  we  follow  the  philosophy  of  Forrester's  Industrial  (or 
Systems)  Dynamics  [12]  in  asserting  that  decisions  should  be  made  about 
each  of  the  flows  indicated  in  fig.  1  and  only  about  these.   We  use  this 
simple  framework  for  assuring  that  the  analyst,  when  interviewing  each 
MIS  user,  has  an  exhaustive  agenda  of  all  required  decisions  In  need  of 
informational  support.   However,  the  costs  and  benefits  of  exhaustive 
agenda  should  not  be  lost  sight  of.   The  analyst  then  works  with  the 
manager  to  model  his  system  in  the  greatest  complexity  possible  in  terms 
of  situational  and  internal  variables  considered  as  candidates  for  inclusion. 


But  he  works  in  the  greatest  simplicity  possible  in  terms  of  the  speci- 
fication of  their  precise  interrelationships. 


Resource 
Allocations 


Fig.  1  Framework  for  decision  modelling 


A  pilot  study  test  of  this  approach  is  discussed  in  section  4  and 
5.   But  before  reviewing  these  we  must  first  digress  J  riefly  to  consider 
the  measurement  of  information  value  since  this  is  central  to  the  inter- 
pretation of  our  results. 
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3.   A  PRACTICAL  APPROACH  TO  INFORMATION  EVALUATION 

Such  properties  as  timeliness,  relevance  and  accuracy  help  to  make 
information  more  valuable,  and  direct  mapping  from  measures  of  these 
properties  to  an  indicator  of  dollar  value  is  possible.   There  seems,  in 
fact,  to  be  fairly  general  consensus  that  information  having  above  prop- 
erties obtains  its  value  from  the  improvements  in  decision  making  which 
it  causes  and  that  its  value  is  to  be  measured  by  the  incremental  profits 
(or  other  indicators  of  goal  accomplishment)  caused  by  these  improved 
decisions.   In  estimating  the  value  of  proposed  new  types  of  information 
the  approach  generally  taken  (see,  for  instance,  [11])  is  some  variant  of 
Bayesian  analysis.   In  order  to  estimate  the  data  required  for  this  approach 
a  number  of  authors  (see,  for  instance,  [7])  have  proposed  the  use  of 
simulation. 

Despite  its  obvious  intellectual  appeal  we  have  found  this  approach 
pragmatically  infeasible  for  three  reasons.   First,  as  Ackoff  [2],  Simon 
[23]  and  Anthony  [3]  have  pointed  out:  information  is  not  merely  used 
for  decision-making,  it  is  also  used  for  such  diverse  but  important 

activities  as  training  and  motivating  subordinates,  problem- finding  and 
strategic  policy-setting;  and  there  is  no  direct  linkage  between  messages 
and  performance  in  any  of  these  cases  since  the  relationship  proceeds  via 
complex  chains  of  intermediaries  and  takes  the  form  of  a  distributed  lag 
function.   Second,  many  of  the  more  important  uses  of  managerial  informa- 
tion are  related  to  ill-structured  problems  which  are  not  well  enough 
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understood  for  Bayesian  modelling  and  simulation.   Where  simulations  are 
forced  on  such  situations  they  distort  and  simplify  reality  to  the  point 
where  their  results  are  invalid  for  information  requirements  specification. 
Third,  in  managerial  information  systems  of  realistic  complexity  the 
same  information  is  typically  used  by  several  managers  and  is,  moreover, 
used  by  each  for  several  different  purposes.   And  to  confound  matters 
still  more,  each  element  of  basic  data  will  typically  give  rise  to 
several  elements  of  disparate  information  as  a  result  of  different  types 
of  processing.   Coping  with  this  complexity  requires  a  far  more  elaborate 
form  of  classif icatory  model  than  has  heretofore  been  proposed. 

Our  approach  is  less  intellectually  appealing  and  is  clearly  in 
need  of  empirical  testing  (to  which  we  are  presently  submitting  it). 
But  it  is,  at  least,  pragmatically  implementable.   Basically,  its 
motivation  is  as  follows.   We  assume  that  managing  and  decision-making 
are,  on  the  whole,  more  important  and  demanding  than  data  preparation. 
Therefore  wherever  it  is  appropriate  to  entrust  decision  responsibility 
for  ill-structured  problems  to  the  intuitive  skill  of  managers,  we 
believe  it  is  a^  fortiori  appropriate  to  trust  the  judgement  of  these  same 
individuals  in  evaluating  their  actual  and  potential  supplies  of  infor- 
mation.  To  do  so,  however,  it  is  necessary  to  remove  bias  and  to 
reduce  all  such  appraisals  to  a  common  scale  of  measurement.   And  it  is 
necessary  to  gather  these  appraisals  in  a  sufficiently  complex  web  to 
be  able  to  evaluate  the  separable  parts  of  a  proposed  system  as  well  as 
its  totality. 


In  reviewing  the  use  to  which  these  appraisals  are  put,  however,  we 
conclude  that  relative  rankings — rather  than  absolute  measures  of  dollar 
value — will  generally  suffice.   This  is  so  because  most  organizations  are 
constrained  by  shortages  of  skilled  systems  people  and  implement  only  a 
fraction  of  the  feasible  and  cost-effective  systems  proposed  to  them. 
A  knowledge  of  absolute  value  is  needed  to  determine  the  optimal  cut-off 
point  in  an  unconstrained  situation;  but  relative  rankings  suffice  under 
a  situation  analagous  to  capital  rationing. 

Consider,  first,  the  problem  of  evaluating  the  information  supplied 
to  a  single  manager.   In  appraising  his  information  needs  we  begin  with 
a  list  of  the  decisions  for  which  he  is  responsible.   To  evaluate  the 
relative  importance  of  the  information  elements  which  serve  him  we  simply 
ask  the  manager  to  rate  each  of  his  decision-types  (on  any  arbitrary 
scale;  say  1-9)  in  terms  of  its  importance  to  the  overall  success  of  his 
job  tenure  and  contribution  to  his  organization.   We  check  to  ensure  that 
these  ratings  take  due  cognizance  of  the  varying  frequencies  of  occurrence 
of  the  different  types  of  decision  (scaling  the  initial  ratings  by  rela- 
tive frequency  if  this  factor  has  not  been  properly  considered) .   Then 
we  normalize  these  ratings  such  that  the  normalized  factors  (across  all 
frequency-adjusted  decision  types)  sum  to  unity.   Next,  we  systematically 
review  each  decision  type  and  list  all  information  elements  which  would 
contribute  to  its  ef f ectivenesss.   In  order  to  evaluate  information,  at 
this  juncture  we  simply  ask  the  manager  to  rate  the  importance  of  the 
contribution  of  each  information  element  to  the  decision  under  review 
(again  on  any  arbitrary  scale) .   Once  again  we  normalize  so  that  the 
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rankings  of  all  information  elements  for  any  given  decision  sum  to  unity. 
We  are  now  in  a  position  to  evaluate  the  contribution  of  each  element  to 
each  decision  as  the  product  of  the  normalized  weight  of  the  decision 
and  the  normalized  weight  of  that  information  element  respectively.   And 
in  cases  where  the  same  information  is  used  in  more  than  one  decision 
we  simply  determine  its  overall  utility  by  summing  the  values  of  its 
several  contributions  as  thus  determined.   From  the  above,  of  course,  it 
will  be  apparent  that  the  total  value  of  all  information  elements  used  by 
this  manager  is  neccessarily  unity.   In  this  sense,  we  have  avoided  bias 
and  have  introduced  consistency. 

For  purposes  of  the  present  study  it  is  not  necessary  to  consider  a 
technique  for  comparing  information  value  across  users.   Suffice  it  to 
say,  then,  that  we  have  devised  a  recursive  procedure  whereby  the  superior 
of  the  base  decision-maker  rates  each  of  the  subordinates  and  personal 
decisions  for  which  he  has  authority.   After  normalization  we  can  then 
determine  a  weighting  factor  to  scale  down  the  Information  value  of  any 
given  subordinate.  Worked  recursively  up  the  organization  to  the  chief 
executive  the  end  result  of  this  process  is  a  system  such  that  the  sum 
of  the  values  of  all  information  elements  is  unity  and  that  the  distribu- 
tion of  this  value  can  be  identified:   by  decision-maker,  by  decision 
type  or  by  information  element  or  sub-element. 

Since  we  were  dealing  with  single  decision-makers  in  our  pilot 
study,  only  the  first  part  of  this  procedure  (excluding  the  recursive 
scaling)  was  used. 


■13- 


Mathematically,  the  process  can  be  described  in  the  following  way: 

Suppose  there  are  "m"  number  of  factors  (or  pieces  of  infor- 
mation or  data)  that  are  affecting  all  the  decisions  of  the 

organization;  f   . .,  f  . 
1       m 

Suppose  there  are  "n"  number  of  decisions  made  in  each  depart- 
ment or  organizational  sub-unit;  D   ...   D 

1 n* 

Suppose  there  are  "r"  number  of  organizational  sub-units  or 

departments  in  the  system;  d  ,     d 

1    * '   r' 

Then  in  general  f.^^^  denotes,  information  element  'i'  affects 

(or  is  needed  for)  decision  "j"  in  department  'k'  where 

i  =  1,  ...,  m 

J  =  1,  ....  n 

k  =  1 r 

Let  Vji^'  represent  the  frequency  of  decision  "j"  in  department  "k". 

^^^    '^jk'  ^^  ^^^  importance  (rank)  of  factor  'i'  in  decision  "j" 
in  department  "k". 

Let  "3^j^"  be  the  importance  (rank)  of  decision  "j"  in  department  "k". 

Let  'Yj^-  be  the  importance  (rank)  of  department  "k"  in  the  system. 

Then  the  importance  (rank)  of  factor  "f   "  for  decision  "j"  in 
department  "k" 

and(BS^).,=    PjkVPjk^Jk 

Therefore,    the   importance    (rank)   of   the   information  element   "f  "   in 
the   total  system 


k  =  1  j  =  1  ^^Jk  •  (?-)jk  -Vk) 

L-1  j§i         ^^^ 

Vi    ^^^y>  i  =   1,    ...,   m 
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From  the  preceding  equations,  it  is  apparent  that  ^y — the  total 
importance  of  all  information  elements  summed — must  equal  unity  times 
the  number  of  departments.   From  this  it  follows  that  each    is  a 
measure  of  relative  importance.   Also  the  sum  of  the  values  of  those 
information  elements  now  provided  can  easily  be  interpreted  in  terms 
of  efficiency  in  meeting  total  (ideal)  information  needs. 

Thus,  the  process  allows  us  to  arrive  at  the  importance  index  (y  ) 
of  all  factors  (information)  we  need  to  keep  in  the  MIS.   Depending 
on  the  budgetary  constraints  and  the  computed  ranks  of  all  the  infor- 
mation elements,  the  inclusion  or  exclusion  from  the  data  base  of  MIS 
may  be  determined. 
A.  Pilot  Study 

The  School  of  Management  of  Case  Western  Reserve  University  was 
the  organization  selected  for  our  pilot  study.   Partly,  of  course, 
this  was  for  the  sake  of  ease  and  economy.   But  the  main  reasons  were: 
first,  that  the  problems  of  department  chairmen  are  quintessentially 
ill-structured  in  nature,  second,  that  there  are  five  such  chairmen 
each  with  essentially  similar  responsibilities  (thereby  permitting 
some  very  modest  statistical  testing)  and,  third,  that  these  chairmen 
are  an  abnormally  sophisticated  group  of  managers  who  would  be  unusually 
skilled  in  articulating  their  information  needs  and  therefore  more 
likely  to  confirm  the  null  hypothesis  that  our  protocol  is  no  more 
effective  than  a  conventional  interview. 
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Our  test  consisted  of  three  parts.   First  we  identified  the  present 
flows  of  information  (initially  by  reference  to  the  administration  of 
the  school,  subsequently  by  follow-up  with  the  chairmen  themselves)  and 
we  asked  each  chairman  for  criticisms  and  suggestions  in  the  conventional 
manner.   Next  we  ascertained  the  activities  and  resources  for  which 
these  chairmen  are  responsible  (initially  by  reference  to  department 
members,  subsequently  by  follow-up  with  the  chairmen  themselves)  and 
we  used  these  decisions  as  a  basis  for  obtaining  further  suggestions 
from  the  chairmen  of  their  information  requirements.   Finally,  we  collated 
our  results,  reviewed  them  with  each  chairman  and  had  him  evaluate  not 
only  his  own  suggestions  but  also  any  information  requirements  suggested 
by  any  of  his  colleagues  but  initially  overlooked  by  him. 

These  steps,  when  told  simply,  sound  like  a  rather  negligible  piece 
of  work.   In  practice  we  found  it  quite  difficult  and  time-consuming 
even  to  discover  what  reports  were  presently  available.   The  analyses 
of  activities  and  resources  were  even  more  onerous.   Many  hours  spent 
with  each  chairman  in  discussing  the  decisions  for  which  he  was  responsible; 
and  still  more  hours  in  exploring  what  information  would  be  helpful 
in  each  of  these  cases.   Ultimately  some  eighty-eight  items  of  infor- 
mation of  one  type  and  another  were  found  to  be  of  value. 
5.   PILOT  STUDY  RESULTS 

The  findings  of  this  study  are  summarized  in  table  1  (which  has 
data  for  only  four  of  the  five  chairmen  since  one,  being  away  on 
sabbatical,  proved  unavailable  for  participation). 

To  help  the  reader  grasp  the  significance  of  this  table  we  will 
explain  how  it  was  prepared. 
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At  the  final  round  of  interviews  each  chairman  subjectively  assessed 
the  importance  and  frequency  of  each  type  of  decision  made  by  him.   Then, 
for  each  decision,  he  subjectively  evaluated  the  contribution  to  its 
effectiveness  that  would  be  made  by  any  of  the  information  elements  pro- 
posed from  any  source  as  being  useful  in  that  context.   To  prepare  the 
table  we  first  normalized  these  ratings  assuming  independence  so  that 
all  information  use-values  by  information  element  and  summed  so  that 
we  knew  the  total  value  across  all  its  various  uses  of  each  of  the 
eighty-eight  information  elements  which  had  been  cited.   Then,  for  each 
chairman,  we  sorted  the  information  elements  into  categories  indicating 
when  and  how  their  usefulness  had  first  been  recognized  by  him.   These 
values  are  entered  in  table  1;  certain  averages  and  percentages  have  also 
been  added  to  clarify  our  conclusions. 

In  column  1  are  listed  the  totals  of  the  subjective  values  of  all 
those  items  which  are  covered  by  the  present  reporting  systems.   On 
average  the  subjective  value  of  these,  it  will  be  seen,  amounted  to 
only  64.3%  of  the  values  of  all  information  ultimately  discovered. 

In  column  3  are  listed  the  subjective  values  placed  on  those  items 
not  now  reported  but  suggested  by  the  chairmen  when  they  were  asked  to 
criticize  present  systems — in  the  manner  of  the  classical  interview 
technique  ("what  extra  information  would  you  like?") .   These  unguided 
suggestions  were  the  source  of  a  12.6%  improvement. 

In  column  5  are  listed  all  further  suggestions  which  resulted  from 
the  interviews  we  conducted  on  the  basis  of  the  protocol  developed  in 
section  2  following  the  framework  ther  depicted  as  fig.  1.   In  these 
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interviews,  that  is  to  say,  we  first  identified  needed  decisions  from 
an  analysis  of  resources  and  activities  and  then  we  enquired  as  to 
the  information  needed  to  support  each  of  these  decisions.   By  now, 
of  course,  it  vi/as  becoming  much  more  difficult  to  discern  unfilled 
needs  since  the  existing  reports  and  the  unguided  suggestions  should, 
presumably,  have  unearthed  all  the  more  obvious  requirements.   Despite 
this  heightening  difficulty,  however,  it  is  interesting  to  observe 
that  our  protocol  not  only  produced  new  insights  but  that  these  actually 
had  a  greater  incremental  value  than  those  disclosed  by  the  unguided 
interview — 17.0%  verses  12.6%;  this  difference,  moreover,  proved 
statistically  significant  at  the  .01  level  on  the  basis  of  a  paired 
test  the  details  of  which  can  be  found  in  [18]. 

From  column  4  of  the  table  it  will  be  noted,  however,  that  even 
after  this  last  interview,  the  average  of  the  aggregate  information 
value  of  the  requirements  discovered  from  all  three  sources  (the 
existing  reports  plus  the  classical  unstructured  interview  plus  the 
interview  shaped  by  our  DEFINEPAC  protocol)  ammounted  to  only  0.844. 
The  remaining  information  elements  (with  an  aggregate  use  value  of 
0.156)  owed  their  discovery  to  the  pooling  of  ideas  across  the  four 
chairmen.   From  column  4  it  will  be  observed  that  some  of  our  subjects 
were  more  productive  than  others:   chairman  3,  for  instance,  came  up 
with  9  3.3%  of  his  requirements  from  his  own  interviews  alone,  whereas 
chairman  1,  came  up  with  only  76.1%  of  his  needs  and  picked  up  the 
remaining  23.9%  from  his  colleagues. 
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Ignoring  these  interpersonal  differences  we  have  developed  a 
naive  sequential-sampling  type  model   of  this  discovery  process  which 
fits  the  data  of  this  pilot  study  tolerably  well,  as  reported  in  [18]. 
This  model  suggests  that  wherever  a  group  of  managers  have  similar  and 
interdependent  responsibilities  it  will  be  worthwhile  to  interview  several; 
but  diminshing  returns  set  in  fairly  rapidly  and  it  will  not  usually 
be  worth  interviewing  all  of  them.   In  our  pilot  study,  for  instance,  if 
the  fifth  chairman  had  become  available  we  should  have  expected  him  to 
contribute  rather  less  than  a  1%  gain  to  the  overall  discovery  of  infor- 
mation requirements. 

Two  further  conclusions,  though  somewhat  subjective  in  nature, 
deserve  reporting.   First,  when  we  were  assembling  samples  of  the 
present  reporting  system  we  found  that  chairmen  could  not,  in  fact, 
name  all  their  present  information  supplies.   Given  the  volume  and 
heterogeneity  of  these  reports  this  is  scarcely  surprising.   But  it  does 
highlight  the  need  for  a  structured  framework  in  any  interviewing  pro- 
cess.  That  the  framework  we  employed  was,  indeed,  highly  productive 
is  clearly  indicated  by  our  results;  but  there  still  remains  the 
possibility  that  our  "modelling  of  the  unmodellable"  failed  in  its 
attempt  to  avoid  construction.   Did  our  interviews  force  a  well- 
structured  mold  on  what  are  really  ill-structured  problems?   In 


*  We  are  in  the  process  of  refining  this  model  to  allow  for  inter- 
personal differences  in  serendipitous  ability.   After  testing  we 
hope  to  be  able  to  report  its  application  to  committee  problem- 
solving  as  well  as  to  information  requirements  analysis. 
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denying  this  charge  we  must  resort  to  a  second — highly  subjective — 
conclusion.   On  examining  the  list  of  information  requirements  finally 
discovered  we  were  very  struck  by  the  wide  variety  of  their  coverage 
(they  included  data  about  such  obvious  topics  as  student  enrollments, 
school  budgets,  faculty  publication  lists;  but  they  also  called  for 
demographic  data  about  alumni,  personal  data  about  trustees  and  a 
variety  of  other  useful  but  surprising  topics).   We  have  been  unable 
to  quantify  this  impression  but  are  convinced  by  its  qualitative  and 
subjective  "feel"  that  our  protocol  did  not  act  as  a  straightjacket. 
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